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Vladimir Shpilrain 

Abstract. We consider IA-endomorphisms (i.e., Identical in Abelianization) 
of a free metabelian group of finite rank, and give a matrix characterization 
of their fixed points which is similar to (yet different from) the well-known 
characterization of eigenvectors of a linear operator in a vector space. We 
then use our matrix characterization to elaborate several properties of the fixed 
point groups of metabelian endomorphisms. In particular, we show that the 
rank of the fixed point group of an IA-endomorphism of the free metabelian 
group of rank n > 2 can be either equal to 0, 1, or greater than (n — 1) (in 
particular, it can be infinite). We also point out a connection between these 
properties of metabelian IA-endomorphisms and some properties of the Gassner 
representation of pure braid groups. 

1. Introduction 

Let F = F n be the free group of a finite rank n > 2 with a set X = {x{\, 1 < 
% < n, of free generators. Then, let F' = [F, F] be the commutator subgroup of the 
group F. The group M = M n = F/F" is called a free metabelian group. 

Let <p be an endomorphism of the group F given by (p(xk) = Vki 1 < k < n. 
It also induces an endomorphism of the group M in the natural way. We denote 
elements of a free group and their natural images in a free metabelian group by the 
same letters when there is no ambiguity. The same applies to endomorphisms. 

Birman Q and Bachmuth Q have used matrices over group rings to study en- 
domorphisms of a free and a free metabelian group, respectively. 

Birman [|| has given a matrix characterization of automorphisms of a free group 
among arbitrary endomorphisms (the "inverse function theorem") as follows. Define 
the matrix J v = (dj(yi))\<i t j< n (the "Jacobian matrix" of if), where dj denotes 
partial Fox derivation (with respect to Xj) in the free group ring ZF (see ||). Then 
ip is an automorphism if and only if the matrix X, is invertible. 

Bachmuth m has obtained an inverse function theorem of the same kind on 
replacing the Jacobian matrix J v by its image over the abelianized group ring 
Z{F/F'). 

Recently, matrix methods have been used by a number of authors to produce 
new interesting results on endomorphisms of free and free metabelian groups. Umir- 
baev [^] has generalized Birman's result to partial generating systems (so-called 
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primitive systems) of a free group. In |7j and JlC|], similar results are obtained for 
primitive systems of a free metabelian group. In monomorphisms (i.e., injective 
endomorphisms) of a free group are characterized by the property of the Jacobian 
matrix to have left independent rows. 

All these results show a remarkable parallelism between the theory of endo- 
morphisms of a free (or a free metabelian) group and the theory of linear oper- 
ators in a vector space. In this paper, we obtain a matrix characterization of 
IA-endomorphisms with non-trivial fixed points ("eigenvectors") which, although 
is similar to the corresponding well-known characterization in linear algebra, also 
reveals a subtle difference. 

IA-endomorphism of a group G is an endomorphism which is Identical in the 
Abelianization G/G' of the group G. As usual, by the rank of a matrix A over a 
commutative ring S we mean the maximal number of rows of A independent over 
S. 

Theorem 1.1. Let (p be an IA-endomorphism of the free metabelian group M = M n 
given by p{xi) = in, 1 < i < n, and let J° be the corresponding abelianized 
Jacobian matrix. Then: 

(i) det(Jp — I) = 0, where / is the nxn identity matrix. 

(ii) If rank(J^ — I) < n — 2, then ip has a non-trivial fixed point inside the com- 
mutator subgroup M' . 

(iii) If rank{J^ — I) = n — 1, then ip has a non-trivial fixed point inside the 
commutator subgroup M' if and only if [xi , a^]** 1 • [^2 5 ^3] U2 • ••• • [%n-ii ^n]"™" 1 = 
[yi,V2} Ul ■ [V2, Vz] U2 ■ ••• • [y n ~i,yn] Un ~ 1 for some elements u k G Z(M/M'), not all of 
them zero. 

The situation is most subtle when rank{J^ — I) = n — 1. In this case, anything 
can happen! For example, if <p is an IA-endomorphism of the group M2 given by 
<p(x\) = x%s; p{x2) = a^s -1 , then ip has no non-trivial fixed points for any s G 
M' 2 , s / 1 (see Proposition 3.2). On the other hand, if ip is an inner automorphism 
induced by an element g £ M' 2 , then <p has a non-trivial fixed point (outside M^)- 
In both situations, rank(J® — I) = 1 = 2 — 1. 

Although I was not able to distinguish these possibilities "by matrix means" , an 
algorithm for detecting fixed points does exist: 

Theorem 1.2. There is an algorithm for detecting the presence of non-trivial fixed 
points of an arbitrary IA-endomorphism of a free metabelian group M n . Also, there 
is an algorithm for detecting the presence of those non-trivial fixed points that belong 
to the commutator subgroup M' n . In both cases, if an algorithm reveals the presence 
of non-trivial fixed points, then at least one of them can be actually found. 

In case of a free group, these questions remain open even when restricted to 
detecting fixed points of automorphisms. 
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When all non-trivial fixed points of (p are inside the commutator subgroup M' 
(which is the case, for example, when <p is the conjugation by a non-trivial element 
from M'), the fixed point group Fix ip has infinite rank: 

Proposition 1.3. Let <p be an IA-endomorphism of a free metabelian group M. 
Then the intersection of the fixed point group Fix (p with M' is a normal subgroup 
of M. In particular, if this intersection is non-trivial, then it is infinitely generated. 

Furthermore, if (p has the fixed point group of finite rank, then there is a "gen- 
eration gap": 

Theorem 1.4. Let ip be an IA-endomorphism of a free metabelian group M n . If 
the rank of the fixed point group Fix ip is finite, then it is equal to 0, 1, or is greater 
than (n — 1) . 

Then, we have 

Proposition 1.5. For an arbitrary n > 3, there are non-inner (IA-)automorphisms 
of the free metabelian group M n that have infinitely generated fixed point group. 

Note that every IA-automorphism of the group M2 is inner |1J]. Then, if u £ M' n , 
the conjugation by u is an automorphism of M n whose fixed point group coincides 
with M' n and therefore is infinitely generated. The same argument proves the ex- 
istence of inner automorphisms with infinitely generated fixed point group for any 
group of the form F/[R, R] with F/R infinite. 

We should also mention here an example of a finitely presented group || whose 
automorphisms can have infinitely generated fixed point group. 

All this makes contrast to the situation in a free group: fixed point group of any 
free endomorphism has rank n = rank F or less. This has been proved in P] for 
automorphisms, and in Q - for arbitrary endomorphisms. 

In general, it is an interesting and important question - how far is the similarity 
between free and free metabelian endomorphisms extended? The desire to "lift" 
properties of metabelian endomorphisms to those of free endomorphisms leads to 
several interesting questions of which the following one seems particularly attrac- 
tive: 

Problem 1. Suppose ip £ Aut F is an IA-automorphism with a non-trivial fixed 
point. Is it true that <p has a fixed point outside [F, F] ? 

Another interesting question is - how can our Theorem 1.1 be strengthened when 
restricted to automorphisms; in particular, we ask: 

Problem 2. Is it true that every IA-automorphism of the group M n has a non- 
trivial fixed point? 

Informally speaking, endomorphisms with non-trivial fixed points are rare. On 
the other hand, endomorphisms which are automorphisms, are also rare (in contrast 
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to the situation in linear algebra!). Whether or not automorphisms with non-trivial 
fixed points are rare, remains a mystery. 

Finally, we point out a connection between the properties of metabelian IA-endo- 
morphisms described above, and some properties of the Gassner representation of 
pure braid groups. For defenitions of braids, braid groups, and for a background 
material, we refer to (4j. 

Let a G P n be a pure braid on n strands, and a the corresponding link. Then, 
let r n _i(<r) be the image of a under the reduced Gassner representation of the 
pure braid group P n , and A^(tx, ■■■,t n ) the Alexander polynomial of a. Then (see 
[4, Theorem 3.11]): 

A&{h, ...,t n ) = if and only if det(T n ^x(a) - I) = 0. 

To explain how this is connected to the subject of the present paper, we men- 
tion that the pure braid group P n is isomorphic to a subgroup of the group of IA- 
automorphisms of F n , and the Gassner representation maps every IA-automorphism 
in this subgroup onto its abelianized Jacobian matrix. Then, the matrix T n —\{a) 
that corresponds to the reduced Gassner representation, is an (n — 1) x (n — 1) ma- 
trix which is obtained from that abelianized Jacobian matrix by applying a suitable 
conjugation, and deleting the last row and the last column of the form (0, ...,0, 1). 

Therefore, if we denote the free IA-automorphism corresponding (under the 
unreduced Gassner representation) to the braid a by the same letter, the above 
condition takes the form rank(J£ — I) < n — 2, which points to part (ii) of our 
Theorem 1.1. 

Thus, we have: 

Corollary 1.6. The Alexander polynomial of a link a is zero if and only if the 
free automorphism which corresponds to the braid a, has a non-trivial fixed point 
g G F' modulo F" , i.e., a{g) = g {mod F"). 

2. Preliminaries 

Let ZF be the integral group ring of the group F and A its augmentation ideal, 
that is, the kernel of the natural homomorphism e : ZF — > Z. More generally, when 
R < F is a normal subgroup of F, we denote by A^j the ideal of ZF generated by 
all elements of the form (r — 1), r G R. It is the kernel of the natural homomorphism 
e R : ZF^Z{F/R). 

The ideal A is a free left ZF-module with a free basis {{xi — 1)}, 1 < i < n, 
and left Fox derivations di are projections to the corresponding free cyclic direct 
summands. Thus any element u G A can be uniquely written in the form u = 
E?=i di(u)(xi - 1). 

One can extend these derivations linearly to the whole ZF by setting e^(l) = 
di(l) = 0. 
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The next lemma is an immediate consequence of the definitions. 

Lemma 2.1. Let J be an arbitrary right ideal of ZF, and let u € A. Then u € J A 
if and only if di(u) € J for each i, 1 < i < n. 

Proof of the next lemma can be found in || . 

Lemma 2.2. Let R be a normal subgroup of F, and let y G F. Then 
ArA if and only if y € R'. 

We also need the "chain rule" for Fox derivations (see ||): 

Lemma 2.3. Let (j) be an endomorphism of F (it can be linearly extended to ZF) 
defined by (j)(xk) = yk, 1 < k < n, and let v = (p(u) for some u,v € ZF. Then: 

d j( v ) = Efc=iv(4(w))dj(yfc)- 

Lemma 2.3 implies the following product rule for the Jacobian matrices which 
looks exactly the same as in the "usual" situation of analytic functions and Leibnitz 
derivations: if (f and ip are two endomorphisms of F, then 

J<p(ip) = 4>{J<p) ■ J-4>- 

If furthermore (p and tp are IA-endomorphisms, then considering abelianization 
of this product rule yields: 

In particular, there is a (faithful) representation of metabelian IA-automorphisms 
by matrices from GL n (Z(F/F')) - cf. §. 

Another version of Lemma 2.3 is: if u, v E ZF and v = (f(u), then 

(di(v), ...,d n (v)) = ((p(di(u),...,ip(d n (u))) ■ J v . (1) 

Finally, we recall a well-known action via conjugation of a group ring Z(F/R) 
on abelian group R/R' under which R/R' becomes the (left) relation module of the 
group F/R. Namely, if g 6 F/R and r € R, then g acts on rR' by taking it to 
grg~ 1 R'; then this action is extended to the whole group ring Z(F/R) by linearity. 
The result of the action of u € Z(F/R) on rR' is denoted by r u R' . Then (see @): 

Lemma 2.4. For r G R, u £ Z(F/R), one has di(r u ) = u ■ di(r) (mod A^). 
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3. Proofs 



Proof of Theorem 1.1. 

(i) Let if be an IA-endomorphism of the group M; ip(xi) = y\, 1 < i < n. 

Then, by the definition of Fox derivatives, we have j/j — 1 = Y2k=i dk(Ui)( x k — 1) 
for any i , 1 < i < n. 

Since ip is IA, abelianizing this equality gives x" — 1 = 5^fe=i ~~ 1)) 

or, in the matrix form: 

(Or? - 1), (x« - 1))* = J« • ((a;? - 1), « - 1))', 

where * means taking the transpose (i.e., we consider a column, not a row). This 
is equivalent to 

(J£-/)-(K-l),...,«-l))* = 0, 
hence the columns of the matrix (J® — /) are dependent, so det(J^ — I) = 0. 

(ii) Let </? take Xj to XjSj, Sj <G M', 1 < i < n. For notational convenience, consider 
the endomorphism ip being lifted to an endomorphism of F. We shall use the same 
letter <p for this lifted endomorphism. When u € F, we denote the abelianization of 
u (i.e., the image in the group A = F/F') by u a . This agreement also applies to 
elements of the group ring ZF. 

If det(J® — I) = 0, then the rows of the matrix (J^ — /) are dependent over the 
group ring ZA = Z(F/F'). Make a new matrix J' upon multiplying the kth row of 
the matrix (J^ — /) by xjj*, 1 < k < n. It is clear that the rows of J' are dependent 
over the group ring ZA, too. 

Now, J' is the abelianized Jacobian matrix of the endomorphism ip' that takes 
Xj to Sj, 1 < i < n. Indeed, dj(xiSi) = 5ij + Xidj(si), so that Xidj{ip' \xi)) = 
dj(<p(xi)) - S i:j . 

Thus, the abelianized Jacobian matrix J^, has dependent rows (over the group 
ring ZA). We can write this dependence as follows: 

n 

J2uk-d?(s k ) = (2) 
k=i 

in ZA for some u k £ ZA, not all of them 0, and i runs from 1 through n. 

By Lemmas 2.1, 2.2, 2.4, the system (2) is equivalent to the following relation: 

n 

IK fe = 1 

fc=i 
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in the group M. 

If rank J", = m < n — 2, then there are precisely m independent elements (let us 
say, s±, s m ) among (in other words, these si, s m generate a free submodule 
of the relation module of the group A), and for any j, m + 1 < j < n, we have a 
relation of the form 



n °t o) 



k=i 



for some Wj, Vkj G ZA 

Now, we are going to find a non-trivial fixed point g G M' of the endomorphism 
<p, in the form 

9 = [X1,X 2 ] Z1 • [x 2 , X 3 P • ... • [x n _i, XnY™- 1 (4) 

for Zfe G ZA (to be found). 

We need a couple of simple observations: 

Lemma 3.1. (i) Let ip(g) = g for some g € M' . Then tp(g u ) = g u for any 
u G ZA. 

(ii) For any element /i € M', one has h w = g for some u; G and for ^ G M' of 
the form (4). 

Proof, (i) Since is IA, we have ip(u) = u in ZA. The result follows. 

(ii) It is clear that the elements [xj,^-], 1 < % < j < n, generate M' as a normal 
subgroup of M. It is well-known that the elements [a;i,a;2], [x n _i,x n ] generate a 
free submodule (of rank (n — 1)) of the relation module of the group A. On the other 
hand, there are no free submodule of rank n in this relation module. This means for 
any pair i,j of indices, one has [xi,Xj] Vi ' = [x\,X2] Ul ■ [x2,x^} U2 ■ ... ■ 
for some Vij, Uk G ZA. The result follows. 

We continue now with part (ii) of Theorem 1.1. From ip(g) = g, we get 

[si,X 2 ] Zl ■ [x 1 ,S 2 ] Zl ■ ... ■ [Sn-uXn]*"- 1 ■ [x n -i , Sn] 2 "" 1 = 1, 

or, equivalently: 

(£2-1)21 (l-Xi)zi (x„-l)«„_i (l-x n -l)z n -l _ 1 /cr\ 

b 1 6 2 ... 6 n-1 b n — 1. {O) 

n 

Conjugate both sides of (5) by w = Wj, where Wj G ZA come from (3). 

k=m+l 

Then, replace every s ■ 3 , j > m + 1, in (5), with the corresponding element on the 
right-hand side of (3). This gives 

4 ■ ... • & = 1 (6) 
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for some z' k G ZA, each of which is a ZA-linear combination of Z{, 1 < i < n — 1. 

Since we have chosen si, s m so that they generate a free sub module of the re- 
lation module of the group A, the equation (6) is equivalent to a system of equations 

4 = 0, 1 < jfe < m. (7) 

This is a system of m homogeneous ZA-linear equations in (n — 1) > m 
unknowns z\, z n -\. It is well-known that a system like that has a non-trivial 
solution over ZA (since ZA is a commutative domain). This completes the proof. 

(iii) The "if part is obvious. The "only if part follows from Lemma 3.1. 

The following example shows how subtle a situation might be in the case when 
rank(J^ — I) = n — 1. 

Proposition 3.2. In the group M2, let ip take x\ to x±s, X2 to X2S~ 1 for some 
s G M 2 , s / 1 . Then 99 has no non-trivial fixed points. 

Proof. First we show that <p has no non-trivial fixed points in M' 2 . Suppose ip(g) = 
g; g G M' 2 . Since 5 has the form [x l5 X2] z for some z G ZA, we have (cf. (5)): 

[xi, x 2 ] z = [x l8 , x 2 s- l ) z = [x u x 2 ] u ■ s (*2-i)«+(*i-i)*. ( 8 ) 

It follows that 8 (x2-i)z+{xi-i)z = 1; hence (x 2 _ l) . z + ( Xl _ l) . z = 
(a?2 + xi — 2) • z = in the group ring which is only possible if z = 0. Thus, 
5=1- 

Now let (7 G M2 be an arbitrary element. Then g has the form x^x^ [xi, X2] 2 
for some 2 G ZA. Let us see first what the image of x™x 2 looks like. 

We have: ip{x™x 2 ) = x\ ■ s ■ ... -x\ ■ s -X2 ■ s _1 • ••• -X2 ■ s" 1 = h. Now we apply to this 
element h the following "collection process": first we collect all the x\ on the left 
by permuting them with s. This gives: h = x™-s x ^ ■ ... ■ s Xl ■ s ■ X2 ■ s _1 • ... • X2 ■ s^ 1 . 
Then, in the same manner, we collect all the X2 on the right of X™: 



h = x™xV^ ■ s x i x % ■ ... ■ s x i x 2 ■ s x 2 . s x 2 • ... ■ s X2 ■ s 1 , 
or, equivalently: 

_ x ™ x ™ . s x™~ 1 x%+...+xix%+x%-x 2 l ~ 1 -...-X2-l j-g-j 

Now write down the whole image of g: <p(g) = <^{x™x 2 • [xi,X2] 2 ) = h • [x\,X2] z ■ 
s (x 2 -i)z+(x 1 -i)z ^ c £ jf _ g^ fa en combining this with (9) yields: 

s x™~ 1 x%+...+x 1 x™+x%-X2~ 1 -...-x 2 ~l+( x 2-l)z+(x 1 ~l)z _ 1 
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or, equivalently: 

x^xZ + ... + x x x% + x%- x^" 1 - ... - x 2 - 1 + (x 2 + xi - 2) ■ z = 0. (10) 

All we have to do now is to show that w = x™ _1 X2 +...+X1X2+X2 — x^ --1 — x 2 — 1 
is not divisible by v = x 2 + x\ — 2 in the ring ZA. This is easy to see upon setting 
xi = — 1; then w' = (— l) m X2 — x^ -1 — ... — x 2 — 1; v' = x 2 — 3. Since x 2 = 3 
is not a root of the polynomial w', the result follows. This completes the proof of 
Proposition 3.2. 

Proof of Theorem 1.2. Let 93 be an IA-endomorphism of the group M = M n . 

First of all, we compute the rank of the matrix . If it is not equal to (n — 1), 
then we just refer to Theorem 1.1 (i), (ii). 

Suppose rank(J^ — I) = n — 1. To find out if there is a non-trivial fixed point 
of ip inside the commutator subgroup M', we consider a system (7); but this time, 
it is a system of (n — 1) homogeneous ZA-linear equations in (n — 1) unknowns 
zi, ...,z n -±. To find out if it has a non-trivial solution (which happens if and only 
if ip has a non-trivial fixed point inside M'), we just compute the corresponding 
determinant and see if it is equal to 0. 

A somewhat more difficult problem is to find out if there is a non-trivial fixed 
point of ip outside M' . In this case, we proceed as in the proof of Proposition 
3.2, but instead of having just one equation of the form (10), we'll have a system 
of (n — 1) ZA-linear equations (they are no longer homogeneous!) in (n — 1) 
unknowns. 

Again, since ZA is a commutative domain, we can resolve this system (by using 
the Kramer's formula). To apply the Kramer's formula, all we need is to be able 
to find out for a given pair of polynomials, whether or not one of them is divisible 
by another. But this is equivalent to asking whether or not one of them belongs to 
the ideal generated by another one. Algorithms like that do exist (in particular, for 
ZA); they are based on what is known as Grobner reduction process. 

In any case, the existing algorithms in (Laurent) polynomial algebras not only 
tell us whether or not a given system of ZA-linear equations has a solution, but if 
it does, they give a solution (although we may not be able to find all of them). This 
completes the proof of Theorem 1.2. 

Proof of Proposition 1.3. First of all, we note that the group Fix ip n M' 
is abelian; therefore, if it were finitely generated, then its every subgroup would be 
finitely generated, too. 

Now suppose g <G Fix ip n M'; g ^ 1. Then, by Lemma 3.1 (i), g u G Fix ip fl M' 
for any u G ZA (i.e., Fix (p D M' is a normal subgroup of M). We are going to 

k 

show that the subgroup of M generated by all g x i , k G Z, is not finitely generated. 
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By means of contradiction, suppose it is generated by g Xl , ...,g Xl , m > 0. But 
every element from this finitely generated group has a form g p for some Laurent 
polynomial p = p(x±), whose degree does not exceed m. Therefore, we don't have 

m+l 

the element g 1 in this group, hence a contradiction. 

Proof of Theorem 1.4. By means of contradiction, suppose H = Fix ip is a 
non-cyclic subgroup of M generated by h±, h r , r < n. Then there is a generator 
of the group M, say x\, such that x™ £ H ■ M' for any m > 1. 

By Lemma 3.1 (i), the group S = H n M' is normal in M. If is non-cyclic, 
then S 1 is non-trivial since it contains a non-trivial subgroup H' (note that H % M' 
since otherwise, H would be infinitely generated by Proposition 1.3). Let s G S, 
s / 1. Then for any m > 1, we have an equality of the form 

i 

for some integers Ci j>m . Let d^(s) ^ 0; then (by Lemma 2.4) applying the derivation 
cZ£ to both sides of the last equality gives (in the group ring ZA) : 

Z ZT[ h l i ' rn - C i,,rn-dl{h iq ) = xT-dl{s) (11) 
1 3 

for some collection of indices j, q (of no particular importance to us) . 

When m runs through 1 to oo, we are encountering representatives of infinitely 
many distinct cosets of the group H ■ M'/M' (as a subgroup of M/M') in the 
supports of elements on the right-hand side of (11). This is due to the condition 
x™ £ H ■ M' for any m > 1 - see above. (By support of a group ring element 
u G ZG, u = c g ' 9> we mean the set {g, c g ^ 0}). 

At the same time, the collection of coset representatives of H ■ M'/M' in the 
supports of elements on the left-hand side of (11) does not depend on m, and is 
therefore finite. This contradiction completes the proof of Theorem 1.4. 

Proof of Proposition 1.5. Consider an (IA-)automorphism ip of the group M n , 
n > 3, given by p{x\) = x\[x2,x$,X]\; <p(xi) = x«, i > 2. We are going to prove 
that the group Fix ip is infinitely generated. 

It is clear that Fix p contains a subgroup of M n generated by X2,---,x n . Now 
we are going to look for (other) fixed points in the form 

g = x$-[x 1 ,x 2 ] ui ■ [x 2 ,x 3 ] u *-...-[x n - 1 ,x n ] u "-i-x 2 ni -...-x™ n - 1 . 

Starting with <p(g) = g and arguing along the same lines as in the proof of Theorem 
1.1 (ii) and Proposition 3.2, we finally arrive at 

^-i + (xi-i)(x 2 -i)-m = o, 
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which is only possible if k = u\ = 0. 

It follows that x™ Fix if ■ M' for any m > 1, and applying the argument 
from the proof of Theorem 1.4 yields the result. 
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